Fingerprint generation of multimedia content based on a trigger point with the multimedia content

ABSTRACT

Surrogate heuristic identification is described, including a memory configured to store data associated with content, and a processor configured to select a portion of the content and the portion is standardized, to identify a characteristic associated with the portion, to use the characteristic to generate a data representation, the data representation being used to provide heuristic data, and to process the heuristic data to generate a fingerprint configured to compare against one or more stored fingerprints.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is related to co-pending U.S. patent application Ser.No. 11/408,199, filed Apr. 20, 2006 and entitled “Surrogate Hashing,”U.S. patent application Ser. No. 11/824,983, filed Jul. 2, 2007, andentitled “Surrogate Heuristic Identification,” U.S. patent applicationSer. No. 11/824,815, filed Jul. 2, 2007, and entitled “SurrogateHeuristic Identification,” U.S. patent application Ser. No. 11/824,982,filed Jul. 2, 2007, and entitled “Surrogate Heuristic Identification,”U.S. patent application Ser. No. 11/824,996, filed Jul. 2, 2007, andentitled “Surrogate Heuristic Identification,” U.S. patent applicationSer. No. 11/824,924, filed Jul. 2, 2007, and entitled “SurrogateHeuristic Identification,” U.S. patent application Ser. No. 11/824,789,filed Jul. 2, 2007, and entitled “Surrogate Heuristic Identification,”U.S. patent application Ser. No. 11/824,995, filed Jul. 2, 2007, andentitled “Surrogate Heuristic Identification,” U.S. patent applicationSer. No. 11/825,001, filed Jul. 2, 2007, and entitled “SurrogateHeuristic Identification,” U.S. patent application Ser. No. 11/824,963,filed Jul. 2, 2007, and entitled “Surrogate Heuristic Identification,”U.S. patent application Ser. No. 11/824,957, filed Jul. 2, 2007, andentitled “Surrogate Heuristic Identification,” U.S. patent applicationSer. No. 11/824,960, filed Jul. 2, 2007, and entitled “SurrogateHeuristic Identification,” U.S. patent application Ser. No. 11/824,846,filed Jul. 2, 2007, and entitled “Surrogate Heuristic Identification,”all of which are herein incorporated by reference for all purposes.

FIELD OF THE INVENTION

The present invention relates generally to searching using processingsystems. More specifically, surrogate heuristic identification isdescribed.

BACKGROUND OF THE INVENTION

Information on the Internet and World Wide Web is available in variousforms, formats, types, and amounts. The Internet has been a valuablemedium for enabling the proliferation of information. However, locatingcopies of information or data is a difficult task using various types ofconventional techniques.

Some conventional techniques rely upon the use of locating oridentifying keywords or text associated with a given image, document,photo, picture, song, or other digital data file (“file”). Keywords ortext (e.g., metadata) may be associated with a given file to enablesearch engines, crawlers, bots, and other search applications to findfiles based on the keywords or data. However, keywords or text do notnecessarily indicate the actual content of a given file. For example, akeyword “tree” may be assigned and used to identify the picture of ariver. As another example, metadata using keywords describing a publicpersonality may be used to describe a web page associated with acompletely different personality. In other words, conventionaltechniques that use keywords and metadata to locate files are typicallyinaccurate and inefficient.

Other conventional solutions may rely upon the use of identifyingobjects within certain types of files. However, object data within afile may be obscured or modified such that conventional techniques areunable to locate and identify copies of a given file that may beslightly different from each other because of artifacts that are locatedwithin the image, video, audio, or other data.

Thus, what is needed is a solution for identifying data without thelimitations of conventional techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

Various examples are disclosed in the following detailed description andthe accompanying drawings:

FIG. 1 illustrates an exemplary surrogate heuristic identificationsystem;

FIG. 2 illustrates an exemplary application architecture for surrogateheuristic identification;

FIG. 3 is an exemplary process for surrogate heuristic identification;

FIG. 4A illustrates an exemplary data representation used for surrogateheuristic identification;

FIG. 4B illustrates an alternative exemplary data representation usedfor surrogate heuristic identification;

FIG. 5A is an exemplary process for developing a data representation forsurrogate heuristic identification;

FIG. 5B is a further exemplary process for developing a datarepresentation for surrogate heuristic identification;

FIG. 6 illustrates an exemplary image used for surrogate heuristicidentification;

FIG. 7 illustrates an exemplary process for surrogate heuristic videoidentification;

FIG. 8A illustrates exemplary video frames used in surrogate heuristicidentification;

FIG. 8B is a further illustration of exemplary video frames used insurrogate heuristic identification.

FIG. 9 is an alternative exemplary process for surrogate heuristicidentification;

FIG. 10A illustrates an example of an application architecture forsurrogate heuristic identification configured to identify audio data;

FIG. 10B depicts one example in which a standardized sample selector canselect different trigger points;

FIG. 11 depicts one example of an analysis performed by a sampleanalyzer to quantify a characteristic with which audio data can beidentified;

FIG. 12 illustrates exemplary surrogate heuristic identificationapplication architecture;

FIG. 13 illustrates an exemplary surrogate heuristic identificationinterface;

FIG. 14 illustrates an exemplary process for using a surrogate hashingor surrogate heuristic identification interface;

FIG. 15A illustrates an exemplary fingerprint application;

FIG. 15B illustrates an alternative exemplary fingerprint application;

FIG. 16 illustrates an exemplary surrogate heuristic identificationsub-process for generating a fingerprint; and

FIG. 17 is a block diagram illustrating an exemplary computer systemsuitable for surrogate heuristic identification.

DETAILED DESCRIPTION

Various examples or embodiments (“examples”) may be implemented innumerous ways, including as a system, a process, an apparatus, or aseries of program instructions on a computer readable medium such as acomputer readable storage medium or a computer network where the programinstructions are sent over optical or electronic communication links. Ingeneral, operations of disclosed processes may be performed in anarbitrary order, unless otherwise provided in the claims.

A detailed description of one or more examples is provided below alongwith accompanying figures. The detailed description is provided inconnection with such examples, but is not limited to any particularexample. The scope is limited only by the claims and numerousalternatives, modifications, and equivalents are encompassed. Numerousspecific details are set forth in the following description in order toprovide a thorough understanding. These details are provided for thepurpose of example and the described techniques may be practicedaccording to the claims without some or all of these specific details.For the purpose of clarity, technical material that is known in thetechnical fields related to the examples has not been described indetail to avoid unnecessarily obscuring the description.

In some examples, the described techniques may be implemented as acomputer program or application (“application”). The describedtechniques may also be implemented as a module or sub-component ofanother application. Here, the described techniques may be implementedas software, hardware, firmware, circuitry, or a combination thereof. Ifimplemented as software, the described techniques may be implementedusing various types of programming, development, scripting, orformatting languages, frameworks, syntax, applications, protocols,objects, or techniques, including C, Objective C, C++, C#, Flex™, Java™,Javascript™, Ajax, COBOL, Fortran, ADA, XML, HTML, DHTML, XHTML, HTTP,XMPP, and others. Design, publishing, and other types of applicationssuch as Dreamweaver®, Shockwave®, and Fireworks® may also be used toimplement the described techniques. The described techniques may bevaried and are not limited to the examples or descriptions provided.

Techniques for surrogate heuristic identification are described. In someexamples, identifying content within files or data streams (i.e.,collections or logical groupings of data that, when instantiated,rendered, or otherwise processed, result in an image, video, audio,multimedia content, picture, or other type of content) from variouslocations on a network (e.g., the Internet). In some examples, surrogateheuristic identification may be used to identify content such as aportion of an image or video, or an image or video. Content (“content”)describes information that may be experienced and that may besubstantially similar in appearance (i.e., when displayed on a screen ormonitor), but may also be different at a data level. In some examples,“data level” may refer to a level or layer of, for example, the OpenSystems Interconnection (OSI), Internet, or other types of datarepresentations and models used to describe interoperability of datacommunication systems. As an example, a clip from a movie may be contenteven though a data representation may be substantially different thananother file due to, for example, encoding schemes or other contentprovided with the clip. In other examples, surrogate heuristicidentification may be used to identify portions of a file or datastream. Likewise, an entire file or data stream may be identified ifimage, video, or audio (i.e., sound) data is provided that may berendered into image, video, or audio (i.e., sound) form. In someexamples, file or content identification may be performed using ahistogram or other type of data representation (i.e., graphical orotherwise) for a characteristic of an image. In other examples, dataused to plot a histogram may be used for file or content identificationwithout plotting (i.e., generating) the histogram or other datarepresentation (e.g., bar graph, pie chart, and others). Histogramvalues may be used for comparison or stored for future comparison. Otherattributes may be used in conjunction with histogram values for contentidentification. In other examples, histogram values may be used togenerate a series of vectors used to identify the content. These vectorvalues may be used for comparison or stored for future comparison. Otherattributes may be used in conjunction with vector values for contentidentification. In yet other examples, histogram values, vector values,a combination thereof, or other data types and values without limitationmay be used with or without other attributes to generate one or moresignatures or fingerprints (“fingerprint”). Fingerprints may represent afile, data stream, portion of a file, portion of a data stream, asegment of content (such as an image piece, video clip, sound byte,etc.), or the whole content. Fingerprints may be encoded histogramvalues, vector values, vector data, heuristic data, a combinationthereof, or other types of data. Other data may be included infingerprints. Fingerprints may also be hashed (e.g., MD5 hash, andothers) to generate values from histogram data. Hashed values derivedfrom vector data, hashed values derived from other data (such as binarydata), or any combination there of may be used. These fingerprints maybe used for comparison or stored for future comparison. Other attributesmay be used in conjunction with fingerprints for content or fileidentification. For example, another image may have one or morehistograms generated, and the vectors of those histograms may be used togenerate a fingerprint for a given file or data stream, which may befurther used to generate a hash value using the vectors. Hashingalgorithms and techniques may be used to generate hash values, such asthose described in U.S. patent application Ser. No. 11/408,199, filed onApr. 20, 2006, entitled “Surrogate Hashing,” which is incorporatedherein by reference for all purposes. In other words, vectors may beused to index, store, and compare histogram values. Fingerprinting is away to index, store, and compare vectors or histograms. Vectors,histograms, or a combination thereof may be used to perform contentidentification using the techniques described herein. In other examples,identification may be performed for other types of files or content,such as audio, by using various characteristics (e.g., frequencies, highor low tones, volumes, and others) to generate histogram data that maybe used with the techniques described. A characteristic may be anyquantifiable property of an image or a portion of an image that may beresolved into a given value. For example, a characteristic may be thebrightness, color, tone, hue, luminance, red/green/blue value, volume,frequency, tone, pixel density, pixel count, and others for varioustypes of files and data streams. A characteristic may also be the rateor degree of change in an attribute. A histogram may be a graph thatrecords counts of a characteristic at various levels. For example, ahistogram may be generated to indicate a number of pixels having abrightness value of 50 or a number of pixels having a green value of 75.In other examples, a histogram may also be generated to represent changein a characteristic (e.g., brightness, contrast, and others) from, forexample, surrounding pixels or between pixels. A pixel may describe a“picture element” as an abstract sample and is not necessarily thesmallest possible sample (i.e. a dot or square). In other words, a pixelmay represent an abstract sample of a file or content (e.g., image,video, audio, and others), but does not necessarily refer to thesmallest picture element in, for example, an image. Pixel may becalculated from smaller parts by methods that may include, for example,average value, maximum value, minimum value, or other. Multiple imagesmay be normalized so that each image includes the same number of pixels.In some examples, histogram values may be discrete or may be grouped inranges. In other examples, histograms axes may be adjusted (e.g.,logarithmic axis) or normalized for given types of content. In stillother examples, histogram values on modified axis may be adjusted toreflect relative distances. Further, histogram and vector values may becalculated at various significant figures (i.e. 10.0121, 10.012, 10.01,10.0, 10, 0, and others). As another example, fingerprints may be storedin a repository (e.g., database, data mart, data warehouse, storage areanetwork, and the like) and may be later retrieved for use in identifyingcontent, files or data streams. In other examples, the describedtechniques may be varied and are not limited those provided.

FIG. 1 illustrates an exemplary surrogate heuristic identificationsystem. Here, system 100 includes network 102 coupled, indirectly ordirectly, to servers 104-108, repository 110, computing device 112,output data 114, and input data 116. Here, crawlers (not shown) may beimplemented or installed on servers 104-108 and used to search or“crawl” the Internet or another data network (e.g., private networkssuch as local area networks (LANs), wide area networks (WAN), municipalarea networks (MAN), wireless local area networks (WLAN), and others) tofind and identify content, files or data streams. In some examples, whencontent, files or data streams are found, characteristics or attributes(e.g., brightness, color, tone, hue, luminance, red/green/blue value,volume, frequency, tone, pixel density, pixel count, and others) may beused to identify content, files, or data streams using the techniquesdescribed. In some examples, a fingerprint may be used in a searchagainst other fingerprints stored in repository 110, to identify anymatches. Matches may be ranked based upon, for example, similarity tofingerprints used in a search. If a match is found, then a location forthe identified segment of content, file or data stream may be returned.In other examples, if a match is not found, then the fingerprint may bestored, along with a location (e.g., uniform resource locator or otherURL) in repository 110. Further, after developing fingerprints for filesor content found on, for example, the Internet, other fingerprintsstored in repository 110 may be compared in order to determine if thefiles or content being identified is similar. In some examples, when amatch is identified, further actions may be taken to enforceintellectual property rights, licenses, injunctions to prevent furtherillegal dissemination, or the like.

FIG. 2 illustrates an exemplary application architecture for surrogateheuristic identification. Here, application 200 may include logic module202, input module 204, crawler interface (I/F) 206, hash module 208,database system I/F 210, histogram module 212, vector module 214, andfingerprint module 216. Here, logic module 202 may guide the operationof application 200 and the various elements (e.g., input module 204,crawler interface (I/F) 206, hash module 208, database system I/F 210,and histogram module 212, vector module 214, and fingerprint module 216)shown. In some examples, input module receives input via input module204. Data may be sent/received between application 200 and crawlersusing crawler I/F 206. Hash module 208, may be used to run hashingalgorithms against values associated with fingerprints, vectors, orother data associated with a file in order to generate hash values. Insome examples, vector module 214, may be used to generate vector datafrom histogram module 212. Likewise, fingerprint module 216 may be usedto generate fingerprints from hash module 208, histogram module 212, andvector module 214, which may be stored in repository 110 (FIG. 1) usingdatabase system (DBS) I/F 210. Further, histogram module 212 may be usedto generate histograms using data associated with files found bycrawlers. In other examples, other types and forms of datarepresentations (e.g., bar charts, pie charts, and others) may be usedand are not limited to those shown. Different modules may be used toreplace histogram module 212 in order to implement different types andforms of data representations. Further, application 200 and theabove-described elements may be varied and are not limited to thedescriptions provided.

FIG. 3 is an exemplary process for surrogate heuristic identification.Here, a file or data stream is selected (302). A copy of the file ordata stream is retrieved or accessed (304). A characteristic may bedetermined for the type of file, data stream or portion thereof (306).Using the characteristic (e.g., color channel, frequency, pixel data,pixel count, and others), a histogram is generated for the file, datastream, or portion thereof (308). Using the histogram, vectors aredetermined (310). Fingerprints may be generated using histogram data,vector data, other data, or a combination thereof, which is associatedwith a given file, data stream, content or portion thereof (312). Aftergenerating a fingerprint, a determination is made as to whether a searchis performed using the fingerprints (314). In some examples,fingerprints may be compared against other fingerprints stored in, forexample, repository 110 (FIG. 1) (316). In other examples, a search maynot be performed and one or more fingerprints may be stored (318). Inother examples, the above-described process may be varied and is notlimited to the examples provided above.

As an example, image or video content may be identified using theabove-described processes. For example, images may be identified bycreating a fingerprint for each image found on a network (e.g.,Internet, world wide web, private or corporate LAN, and others), database (e.g., repository 110 (FIG. 1)), and others. A fingerprint may begenerated for an image, video, audio, picture, photo, multimedia, orother type of content. Once generated, a fingerprint may be compared topreviously-stored fingerprints. If a match is found with a storedfingerprint, the location or address of the file or data streamassociated with the generated fingerprint may be returned.

A file or data stream may be used to, for example, render an image. Insome examples, a source file may be used to generate multipleinstantiations, objects, or renderings of the image using anencoder/decoder (“codec”) to encode/decode data that, for example, maybe used to generate a bitmap image. For example, an image may be storedin the joint picture experts group (JPEG) format, which may be decodedusing a JPEG decoder. Various types of codecs may be used and are notlimited to any particular type. In some examples, a decoded file or datastream intended for display on a screen or user interface, for example,may be fingerprinted using the above-described process. Images mayinclude any file or data stream that may be rendered on a display,including documents, portable documents, spreadsheets, slideshows,videos, photos (i.e., digital photographs), collections of pixel data,and others. In other examples, the above-described process may beperformed on other types of files (e.g., audio, video, multimedia,multi-frame, and others). Regardless of file or data stream type,surrogate heuristic identification may be performed by selecting astandardized portion of an image, file, or data stream (i.e., encodedheuristic data or heuristic data). For example, a number of images maybe used to generate fingerprints for each image using a portion selectedfrom a consistent location, area, or part of content (i.e.,standardized). In other words, a standardized set of data (i.e., dataset) may be selected from each image and used to create a fingerprint.In some examples, by retrieving portions of content, bandwidth andprocessor requirements may be reduced, allowing faster processing timesand increasing the amount of content, the number of files, or datastreams that may be fingerprinted.

Here, a histogram may be used to determine vectors and, subsequently,fingerprints for content. In other examples, content such as an entirefile, image, video, audio, movie clip, or the like may be selected as astandardized portion. Here, a threshold size for a portion may beestablished. If a file or data stream has an amount of content that isless than or equal to the threshold file size, the entire (e.g.,complete set, file, group, or other collection of) content may be usedas a portion to plot a histogram from which to derive vectors that areused to generate a fingerprint. In other examples, multiple portions maybe selected. Here, each portion may be used to generate fingerprints fora piece of content. A characteristic or attribute of a given type offile or data stream may be used as inputs for plotting points using acoordinate system for a histogram. In other examples, counting thenumber of occurrences of a given characteristic or attribute may be usedto construct graphical or other types of data representations and models(e.g., pie charts, bar charts, and others) other than a histogram. Inother words, different techniques may be used to analyze events,occurrences, incidences, and other characteristics or attributes. Forexample, stochastic techniques may be used to analyze seemingly randomoccurrences of a given characteristic or attribute of an audio file, butwhen plotted using a histogram, a pattern may emerge or vectors may beplotted. In some examples, these may be used to generate a fingerprint.A histogram may be generated by using measurements of the samecharacteristic for every portion of a file or data stream.

In some examples, vectors generated using a histogram may be used tocreate a fingerprint for content (i.e., a portion). In other examples,vectors may be determined using a small amount of data selected from afile or data stream as a standardized portion. In other words, theamount of data in a selected portion may be varied to include a set ofdata (i.e., dataset) chosen from some or all data associated with a fileor data stream. Fingerprints may include all data associated with ahistogram or a portion of a histogram. For example, a histogram may beplotted with ten discrete levels, for example, that represent brightnesslevels 1-10 for pixels in an image. By counting the number of pixelswithin the various brightness levels, the number or “counts” of pixelsmay be used to plot a histogram. In other examples, differentcharacteristics other than brightness may be used. For example, in anaudio file, frequencies and amplitudes associated with a sound wave maybe plotted and used to generate a histogram.

FIG. 4A illustrates an exemplary data representation used for surrogateheuristic identification. Here, histogram 400 may be a histogram of acharacteristic (e.g., visual, aural, imagery, multimedia, and others)associated with a segment (i.e., portion) of content. In some examples,a pixel count for an image may be used as a characteristic to providedata for plotting points using, for example, a Cartesian coordinatesystem, as indicated by axis 404. Each full or partial (i.e., portion)segment of content used for surrogate heuristic identification may havea histogram, vectors, fingerprint, or a combination thereof. Histogram400 may represent various levels (e.g., 0 to 255) along axis 402 and apixel count along axis 404. For example, there may be approximately ten(10) pixels having a brightness (i.e., intensity) value of zero in theimage or a portion of the image represented by histogram 400. Line graph406 may show the relative intensity level of a predetermined visualcharacteristic along axis 402. For example, line graph 406 mayillustrate a selected portion having a high number of pixel counts inapproximately the 180 intensity range.

In some examples, histogram 400 may be used to develop a fingerprint fora portion of content. Fingerprints from one or more selected portionsmay be used to identify an image, video, audio, multimedia, or othertype of content. Further, fingerprints may be compared against otherstored fingerprints to determine whether the content undergoingsurrogate heuristic identification is substantially similar to othercontent. In some examples, vectors 408-418 may be used to derive valuesassociated with a given item of content. Vectors 408-418, troughs420-428, and peaks 430-440 may be used to perform vector analysis usingvarious types of techniques to derive one or more fingerprints for agiven segment of content. In other examples, the above-describedtechniques may be varied and are not limited to those shown anddescribed. Further, different data representations other than histogramsmay be used and are not limited to the examples shown and described.

FIG. 4B illustrates an alternative exemplary data representation usedfor surrogate heuristic identification. Here, histogram 450 includesaxes 452-454, line graph 456, vectors 458-468, troughs 470-478, andpeaks 480-490. In some examples, histogram 450 may be a histogram of acharacteristic (e.g., visual, aural, imagery, multimedia, and others)associated with a given segment of content. Histogram 450 may be used toperform vector analysis on vectors 458-468. In some examples,fingerprints may be developed by using data associated with histogram450 and each of vectors 458-468 (e.g., angle (i.e., direction),magnitude, and the like) as inputs to surrogate hashing techniques suchas those described above. Other data may be used in conjunction withhistogram data and vectors. With regard to FIGS. 4A and 4B, tolerancesmay be adjusted by rounding vector data (e.g., angle measurements,magnitudes, and the like). As an example, if a given vector has an angleof 30.77 degrees and a magnitude of 10.25, these vector data values maybe rounded to an angle of, for example, thirty-one (31) degrees and amagnitude of 10, thus increasing vector analysis and surrogate hashingtolerances while reducing data storage and processing requirements. Inother words, higher degrees of accuracy of vector data and analysis maybe reduced in order to reduce storage and processing requirements. Usingsurrogate hashing techniques, fingerprints may be generated as a number,set of characters, or as machine-readable data that substantiallyuniquely identifies a given segment of content. Once generated,fingerprints from one or more selected portions may be used to identifyan image, video, audio, multimedia, or other type of content. Asdescribed above, line graph 456 may be resolved into a set of vectors(e.g., vectors 458-468) using troughs 470-478 and peaks 480-490. Dataassociated with vectors 458-468 and histogram values may be used asinputs to surrogate hashing techniques (e.g., algorithms, formulae, andthe like). Further, data may be measured in terms of intensity (i.e.,x-axis) and pixel counts (i.e., y-axis). In terms of intensity,intensity levels may be grouped (e.g., 20-30, 90-100, 110-120, andothers) or placed in “buckets” along axis 452. Buckets, in someexamples, may be pre-determined groups, sizes, or quantities of a givenmeasurements found along axis 452-454. For example, a bucket may includeintensity levels 20-30, 90-100, 110-120, and the like. A bucket may alsobe used to describe groupings of pixel counts (not shown) along axis454. In yet other examples, pixel sizes and sampling rates may bemodified so that content that differs substantially in size, quality, orother factor may be compared. In other examples, pixel counts may alsobe individually indicated or grouped along an axis. In still otherexamples, axes 452-454 may be implemented differently. For example,minimum (e.g., 0) and maximum (e.g., 255) values may be chosendifferently and are not limited to the examples shown. In some examples,the axis of the histogram may be normalized or other scales may be used(e.g., logarithmic scales). Point values used in a histogram may beadjusted to accurately represent a position against a standard axis. Ifgroupings of measurements are used, normalization techniques may be usedto reduce or eliminate distortion along axes 452-454. In other words,data associated with vectors 458-468 may be normalized to allowgroupings as well as individual measurements to be used withoutdistorting histogram 452. As another example, brightness channel (i.e.,intensity) values may be increased or decreased. Yet another example mayinclude shifting values left or right (i.e., for intensity values) or upand down. In other examples, data may be evaluated using techniquesother than vector analysis and using histogram 452. Further, histogram452 may be implemented differently and is not limited to the examplesprovided.

FIG. 5A is an exemplary process for developing a data representation forsurrogate heuristic identification. Here, a histogram is used as a datarepresentation for evaluating selected portions of content for surrogateheuristic identification. In some examples, a level is selected (500).In some examples, a level may include, for example, a bucket for a valuealong axis 402 (FIG. 4A) into which counts are collected. For example,the level may be an intensity of 100, at which there may be 20 counts.The level to the left (i.e., along axis 402 (FIG. 4A) of the chosenlevel is examined (502). For example, as shown here, the level with theintensity of 99 may have 25 counts. The level to the right of the chosenlevel is examined (504). In this example, the level with the intensityof 101 may have 25 counts. A determination is made as to whether thecounts of the levels to the left and the right of a chosen level areboth greater or less (506). If the counts of the levels to either sideof the chosen level are both greater (i.e., suggesting the chosen levelis a minimum), then the chosen level is identified as a minimum (508).If the counts of the levels to either side of the chosen level are bothless, then the chosen level is identified as a maximum (510). Adetermination is made as to whether there are any more levels to examine(512). If there are more levels to examine, another level is selected(500). The above-described process may be varied in design, order, andimplementation and is not limited to the examples provided.

FIG. 5B is a further exemplary process for developing a datarepresentation for surrogate heuristic identification. Here, an extreme(e.g., an absolute maximum or absolute minimum) is selected (520). Adetermination is made as to whether other extreme values (“extremes”)are nearby or in substantially close proximity (e.g., an adjacent peakor trough) (522). If other extremes are nearby, then anotherdetermination is made as to whether the nearby extremes have a greatermagnitude than the selected extreme (524). If the nearby extremes have agreater magnitude than the selected extreme, then the selected extremeis not identified as a global extreme (526). If the nearby extremes arenot greater in magnitude than the selected extreme, then it isidentified as a global extreme, which may also be the determination ifno other extremes are nearby (528). Another determination is made as towhether there are other extremes of the same type (e.g., if the chosenextreme is a minimum, whether there is another minimum) within a certaindistance (i.e., nearby) from the chosen extreme (530). For example, themaximum 436 is within 40 levels of the maximum 434 (FIG. 4A). If thereare no other extremes, then vectors may be determined, as describedbelow (532). However, if other extremes are nearby, a determination ismade as to whether the extreme has a greater magnitude than the selectedextreme (e.g., if a maximum is higher or a minimum is lower). Forexample, the magnitude of maximum 436 (FIG. 4A) may be greater than themagnitude of the maximum 434 (FIG. 4A). If the magnitude of the extremeis greater, the selected extreme is not a global extreme. For example,maximum 434 is not a global extreme because maximum 436 is nearby andhas a greater magnitude. Alternatively, maximum 436 is a global extreme.If other extremes do not have a greater magnitude, then the extreme ischosen as a global minimum or maximum. If there are more extremes,another extreme is selected (520).

If there are no more extremes, vectors may be determined (532). In someexamples, vectors may be determined by finding a distance and anglebetween each adjacent point, each adjacent extreme, or each adjacentglobal extreme. For example, vector 410 (FIG. 4A) may be determined byfinding the distance and angle between maximum 430 and minimum 422 (FIG.4A). Maximum 430 may represent 50 counts at 30 units of intensity, andminimum 422 may represent 20 counts at 100 units intensity. Vector 410may be determined, for example, as arctan

$\left( \frac{- 30}{70} \right) = {{- 23.2}{^\circ}}$from axis 402. The magnitude of vector 410 may be determined, forexample, as √{square root over (70²+(−30)²)}=76.2 using, for example,the Pythagorean Theorem.

In some examples, angles and magnitudes of vectors may be stored andused, for example, as fingerprints of selected content or segment ofcontent (534). For example, a fingerprint for vector 410 may be“−232762,” indicating that vector 410 is −23.2° from the vertical andhas a magnitude of 76.2 units. In other examples, vectors may be hashedusing techniques such as those described above. Further, hashing vectorsmay be useful where many vectors for a single image, file, data stream,or other content are stored.

In some examples, more or fewer significant figures of the fingerprintmay be stored to increase or reduce the tolerance of surrogate heuristicidentification. In other words, a fingerprint of vector 410 may bereduced to a single significant figure and stored as “−2080”, indicating−20° from the vertical and 80 units of length. When searching for animage using the fingerprints, a reference image with fewer significantfigures may increase the tolerance of the image identification. In otherexamples, the techniques described above may be applied to otherhistograms without limitation (e.g., FIG. 4B). In other examples, theabove-described identification process may be varied and is not limitedto the descriptions provided.

FIG. 6 illustrates an exemplary image used for surrogate heuristicidentification. Here, image 600 includes portions 602-608, each of whichmay be selected as a standardized portion to perform surrogate heuristicidentification. A visual characteristic for image 600 may be chosen. Insome examples, a visual characteristic may be brightness, intensity,red/green/blue channel values, contrast, pixel count, pixel density, andothers. Visual characteristics may be used to generate a histogram, asdescribed above in connection with FIGS. 4A-4B. Any number, location,size, percentage, or shape of portions 602-608 may be selected and arenot limited to the examples shown. As an example, portions (e.g.,portion 602-608) may be standardized by selecting the same set, part,sub-part, size, location, percentage, or shape of data from every imageused to perform surrogate heuristic identification. Another portionhaving substantially the same location, size, shape, or othercharacteristics may be selected and used to identify whether the file ordata stream corresponding to the portion is a copy. In some examples,brightness of picture elements (i.e., pixels) may be used to select agiven portion in image 600. In other examples, portions 602-608 may beselected from each image (e.g., image 600) using a predetermined size.In still other examples, portions 602-608 may be selected using acombination of characteristics. For example, portion 602 may be an areain which at least fifty percent of the pixels have a brightness over175. In still other examples, portions of other images (i.e., files ordata streams) may be used to select another standardized portion.Standardized portions are portions that are selected using the samecharacteristic or set of characteristics. For example, portions locatedin the lower right corner of each image are selected. As anotherexample, portions may be selected based on having at least fifty percentof pixels with a brightness level over 75. Once selected, portions maybe used for surrogate heuristic identification, as described above.

FIG. 7 illustrates an exemplary process for surrogate heuristic videoidentification. Here, a visual characteristic is identified (702). Aportion of an image, as described above, is selected (704). Using theidentified visual characteristic (e.g., brightness, R/G/B color channel,pixel density, pixel count, and others), histogram data may be developed(705) using the above-described techniques. In some examples, vectordata is developed (706) using the above-described techniques. Differentdata representations or models may be used and are not limited tohistograms. Identifying the global and local maximums and minimumsallows vector data to be determined. Fingerprints may be generated byencoding various histogram data, vector data, a combination thereof, orother types of data. Fingerprints may also be generated by hashing datausing the techniques described and incorporated by referenced above.Additional data may be encoded into the fingerprints (708). In otherexamples, the above-described process may be varied and is not limitedto the sub-processes, design, order, or implementations described above.

FIG. 8A illustrates exemplary video frames configured for surrogateheuristic identification. Here, frames 802-810 are indexed at timeperiods or intervals (“intervals”) 812-820. In some examples, frames802-810 may be of a video, multiple-framed image, or other type of fileor data stream that has a temporal component (e.g., slide show,presentation, multimedia content, rich internet application, and thelike) or that may be organized into discrete frames and which may beindexed at, for example, intervals 812-820 (e.g., 1:00, 2:00, 3:00,4:00, 5:00 minutes). Frames 802-810 may be used to identify a givenvideo or multiple-framed image. In some examples, frames 802-810 may bea subset of the frames selected from the video at predetermined times.Alternatively, the number of frames and time intervals may be varied andare not limited to the examples shown and described.

FIG. 8B is a further illustration of exemplary video frames used insurrogate heuristic identification. Here, frames 832-840 at intervals842-850 of a file or data stream (e.g., video, audio, multimedia, andthe like) are shown. In some examples, frames 832-840 may be a subset offrames selected from a video file or data stream. The frames may beselected based on one or more characteristics, such as those describedabove. For example, frames 832-840 may be selected because they are thefive brightest frames in a portion of video. As another example, frames832-840 may be selected based on an audio track of the video. Frames832-840 may be selected at various intervals (e.g., intervals 842-850)at which, for example, audio levels (e.g., frequencies, amplitude, andothers) are at their highest level. In other examples, frames 832-840may be selected by determining if audio characteristics are above agiven threshold (e.g., volume, frequencies, amplitudes, and others). Instill other examples, portions that are ignored may have no audiblesounds, thus indicating a higher probability that identifiable images(e.g., black, solid, white, blue, green, or otherwise unused frames suchas those found at the beginning or end of a movie) may not be present.Further, some portions of a video or multiple-framed image orpresentation may have sound (e.g., dialogue, sound track, music, andothers), thus indicating that images may be present that can be used forsurrogate heuristic identification. The above-described examples may bevaried and are not limited to the descriptions provided.

FIG. 9 is an alternative exemplary process for surrogate heuristicidentification. Here, a portion of content is selected (902). Forexample, a portion may have a duration of five minutes (e.g., FIGS.8A-8B). A characteristic may be determined and used for selecting frameswithin a portion (904). A portion may be identified using variouscriteria (e.g., first five minutes, last three minutes, brightest fiveminutes, loudest two minutes, portion of video with greatest motion, andthe like). Characteristics may be determined automatically,systematically, manually, or pre-determined. In other examples, content(e.g., video) may be used entirely (i.e., all of the content, video,audio, image, movie, and others). Once determined, a characteristic maybe used to select frames from the portion (906). In some examples, acharacteristic may be a parameter that specifies that a frame isselected every minute, thirty seconds, ninety seconds, one hundred andfifty seconds, or some other periodicity. Characteristics may also bedifferent than time. Frames may also be selected based on changes thatoccur. For example, the five frames with the most change from theprevious frame may be selected. Color, motion, brightness, luminosity,and other visual attributes of a video may be used. As other examples,the five brightest frames, three dimmest frames, seven reddest frames,or ten frames with the most change from the previous frame may beselected. Frames may be selected because an object is presented in agiven frame, a sound is associated with a given frame or the like. Eachframe selected may be analyzed and catalogued as a separate segment ofcontent. Once selected, frames may be analyzed using the processesdescribed above to generate fingerprints. Fingerprints may be generatedusing techniques such as those described and referenced above. Oncedetermined, fingerprints may be compared to determine whether a copy hasbeen identified and located.

Alternatively, other multiple-frame files or data streams may beidentified using the techniques described. For example, a slideshow,word processing document, multimedia application, webpage, and othersmay be represented as images. Portions may be selected from amultiple-frame file or data stream using various characteristics todetermine which frames are selected. Further, after generatingfingerprints for selected portions (i.e., files or data streamsundergoing surrogate heuristic identification), stored fingerprints maybe used (i.e., compared) to determine whether a match exists. If a matchis found, then the location of the duplicate image, video, or other fileor data stream may be determined, returned, stored, processed, orotherwise used.

FIG. 10A illustrates an example of an application architecture forsurrogate heuristic identification configured to identify audio data.Here, application 1000 may include logic module 1002, input module 1004,crawler interface (“I/F”) 1006, and database system I/F 1010, all ofwhich can include structure and/or functionality as similarly-namedelements in FIG. 2. Further, application 1000 can optionally includesample representation compression module 1008 and/or pattern generatormodule 1012. Sample representation compression module 1008 can beconfigured to compress an amount of data that represents, and, thus,identifies a standardized sample, a file, or the like. As such, samplerepresentation compression module 1008 generates a compressedrepresentation of data, such as audio data. In one embodiment, samplerepresentation compression module 1008 is configured to operate as hashmodule 208 (FIG. 2) to generate hash values as compressedrepresentations of samples or fingerprints, based on vectors, or otherdata associated with a file. In at least one instance, the compressedrepresentation (e.g., fingerprint data) can be stored in repository 110(FIG. 1) using database system (DBS) I/F 210. Note that samplerepresentation compression module 1008 is not limited to implementinghash module 208 to generate compressed representations. Patterngenerator module 1012 can be configured to detect patterns within astandardized sample, as well as generate patterns that can distinguishone standardized sample from other standardized samples. In oneembodiment, pattern generator module 1012 can be configured to operateas a histogram module 212 (FIG. 2) to generate patterns, as described byhistograms, to determine vectors constructed from data associated with,for example, files found by crawlers. Note that pattern generator module1012 is not limited to implementing histogram module 212 to generatepatterns.

In the example shown, logic module 1002 can optionally includestandardized sample selector 1020, an audio detector, and a sampleanalyzer 1022. Standardized sample selector 1020 can be configured toselect triggers points at which a standardized sample can be obtained. Atrigger point, at least in some embodiments, can refer generally to avalue for one or more characteristics and/or attributes that causesapplication 1000 to capture audio data as a standardized sample. Atrigger point can be a high-valued characteristic, a low-valuedcharacteristic, or a range of values that, when detected, will causeapplication 1000 to sample audio data. Examples of characteristicsinclude frequencies, volumes, and other aural attributes that can bepresent in the audio. As such, a trigger point can be a frequency of anaudio signal that meets or exceeds, for example, a specific frequency(e.g., a high frequency). In one embodiment, a trigger point can be thestart of the audio file (e.g., zero minutes and zero seconds ofplayback). FIG. 10B depicts one example in which standardized sampleselector 1020 selects different trigger points.

Sample analyzer 1022 is configured to analyze a standardized sample toassociate one or more quantities for one or more characteristics of thestandardized sample. In particular, sample analyzer 1022 can operate todescribe audio data in a manner that facilitates, for example, thedetermination of patterns with which to distinguish unique standardizedsamples from each other. In one embodiment, sample analyzer 1022 candecompose audio signals into any number of ranges, such as frequencyranges, for characterizing those ranges. FIG. 11 depicts one example ofan analysis performed by a sample analyzer to quantify the features of acharacteristic with which audio data can be identified.

Audio detector 1021 is configured to detect whether data (in a file orstreaming over a network) is audio. In one case, audio detector 1021 candetect audio data as a function, in whole or in part, of a fileextension that indicates an audio or video file (e.g., .wmv, .mp3, etc.,indicate files). In another case, audio detector 1021 can detect audiodata as a function, in whole or in part, of a MIME type (e.g., type:Audio, such as “audio/mpeg”) associated with the data and/or file.Further, logic module 1002 may guide the operation of application 1000and the various elements (e.g., input module 1004, crawler interface(I/F) 1006, database system I/F 1010, sample representation compressionmodule 1008, and pattern generator module 1012) shown. In some examples,input module receives input via input module 1004. Data may be sent orreceived between application 1000 and crawlers using crawler I/F 1006.In other examples, application 1000 and the above-described elements maybe varied and are not limited to the descriptions provided.

FIG. 10B depicts one example in which a standardized sample selector canselect different trigger points. In the example shown, standardizedsample selector 1020 (FIG. 10A) can be configured to select thebeginning of the file 1053, a low-valued characteristic 1058 (e.g., alow frequency), and a high-valued characteristic 1062 (e.g., a loudnoise) as triggers points at which standardized sample 1056,standardized sample 1060, and standardized sample 1062 can respectivelybe obtained. As shown, diagram 1050 depicts the magnitude for acharacteristic being sampled as Y-axis 1052 in relation to time alongX-axis 1054. Application 1000 (FIG. 10A) can capture standardized sample1056 in response to sampling an audio from the beginning of the file (orat receipt, if streaming) to a point 1057 (e.g., the first 2 to 5minutes). Standardized sample 1060 can be captured in response todetecting audio signal 1059 exceeding a low-valued threshold. Oncedetected, application 1000 can obtain standardized sample 1060 during atime interval 1071 in association with low-valued characteristic 1058 asa trigger point. Standardized sample 1064 can be captured in response todetecting audio signal 1063 exceeding a high-valued threshold. Oncedetected, application 1000 can obtain standardized sample 1064 during atime interval 1073 in association with high-valued characteristic 1062as a trigger point.

FIG. 11 depicts one example of an analysis performed by a sampleanalyzer to quantify a characteristic with which audio data can beidentified. In the example shown, sample analyzer 1022 (FIG. 10A) can beconfigured to select standardized sample 1100 of audio for analysis.Audio signal 1101 is configured to include audio. As shown, the sampleanalyzer (not shown) is configured generate a number of characteristicranges 1102, such as ranges R0 to R10. In some cases, each of thesecharacteristic ranges 1102 relate to a unique range of frequencies. Togenerate a pattern, the sample analyzer analyzes audio signal 1101against characteristic ranges 1102 to determine, for example, a timecount 1130 and/or a range-crossing count 1132. A time count 1130indicates a number of time units that a rising edge and a falling edgeof audio signal 1101 encompass an amount of time for a specific range.For example, audio signal 1101 encompasses an amount of time 1104 a andanother amount of time 1104 b, which is shown to be 42 milliseconds(ms). A range-crossing count 1132 indicates a number of times that audiosignal 1101 crosses (e.g., fully crosses) a characteristic range 1102.For example, audio signal 1101 crosses range R4 at 1106, at 1108, and1110, all of which add up to 3 range crossings (as shown). In somecases, time count 1130 and/or a range-crossing count 1132 form the basesfrom which to generate patterns, as derived by, for example, histograms.In other examples, the above-described techniques for analyzingstandardized sample 1100 may be varied and are not limited to thedescriptions provided.

FIG. 12 illustrates exemplary surrogate heuristic identificationapplication architecture. Here, application 1200 includes logic module1202, interface module 1204, query generator 1206, fingerprintevaluator/loader module 1208, fingerprint tables 1210, and applicationprogramming interface (API) 1212. In some examples, application 1200 maybe used to implement the described techniques for surrogate heuristicidentification. As an example, logic module 1202 provides control andmanagement functions for application 1200 and its associated elements(e.g., interface module 1204, query generator 1206, fingerprintevaluator/loader module 1208, fingerprint tables 1210, API 1212). Inother examples, application 1200 and the described elements may bevaried and are not limited to the number, type, function, design,architecture, or structure of those shown.

In some examples, input may be received by interface module 1204. Input(e.g., fingerprints, files, data streams, portions of files or datastreams, and others) may be used to construct a search by querygenerator 1206. Once constructed, a SQL query may be run againstfingerprints stored in repository 1210. Here, fingerprintevaluator/loader module 1208 may be used to compare fingerprints inputby, for example, a user via a user interface. In other examples,fingerprint evaluator/loader module 1208 may be used to comparefingerprints provided by a system or other automatic or semi-automaticinput. Further, fingerprint evaluator/loader module 1208 may be used tocompare fingerprints against stored fingerprints in fingerprint tables1210. When a fingerprint is used to identify a file using the techniquesdescribed above, a location, address, web page, URL, or otheridentifying data may be returned via API 1212. In still other examples,application 1200 and the above-described elements may be varied infunction, structure, and implementation and are not limited to thedescriptions provided.

FIG. 13 illustrates an exemplary surrogate heuristic identificationinterface. Here, interface 1300 includes window 1302, panels 1304-1316,and address bars 1318-1320. In some examples, interface 1300 may be usedas a “front-end” display or interface for application 1200 (FIG. 12).For example, panel 1304 may be implemented to display a web pageassociated with an address (e.g., URL) provided in address bar 1320.Panel 1306 may be used to display a target (i.e., a domain on which agiven file or data stream is found), which may also be identified by anaddress (e.g., URL) displayed or provided (i.e., input into) in addressbar 1318. Panels 1306 and 1304 may follow links presented in the display(e.g., hyperlinks displayed on web pages), which may be implemented ashyperlinked text or images 1322-1324 having an associated link. Domainson which a given file or data stream is found may also be listed inpanel 1308. Address bar 1318 and 1320 also may be used to input a domainfor study. If a given domain is a target, then 1308 may list domainsthat contain similar files to the target domain. Further, web page linksfor domains found using the described techniques may be implementedusing panel 1310. Panels 1308 and 1310 may be used to provide or inputdata. Panels 1308-1310 may also be configured to present data asclickable (i.e., enabled for user interaction when, for example, usingan input/output device (e.g., computer mouse) to interact with anon-screen icon, address, link, or other interactive element) hypertext.In some examples, panel 1312 may be used to display or input ratingsprovided by a user of interface 1300. For example, if a user issearching for a given file or content and application 1200 (FIG. 12)returns domains listed in panel 1308, the user may rate each resultusing various techniques. In some examples, ratings may benumerically-based. In other examples, ratings may be qualitative inputs(e.g., copy, not a copy, close, authorized, licensed user, and the like)provided by a user. Panel 1314 may also be used to provide inputs suchas providing a numerical range of results, geographic limitations (e.g.,based on an IP address), MIME type or other file determinantinformation, domains, fingerprints, reference IDs, row IDs (i.e., forlookup in a table, for example, stored in fingerprint tables 1310 (FIG.13)) and others. In some examples, information and data input in panels1304-1320 may be used as filters for searching (e.g., constructing andrunning queries, and the like) for a given fingerprint, file, or datastream.

In some examples, panel 1305 may be configured to display files 1309,which may be associated with a given website. A website may be, in someexamples, a logical grouping of locations, files, or other content foundon a website, which may be targeted for the purpose of identifyingcontent using the described techniques. Each of files 1309 may beselected or deselected by, for example, checking or unchecking a boxnext to each file name. File names may be listed on a line next to eachbox. File names may be selected or deselected using different techniquesand are not limited to the examples provided.

Panel 1307, in some examples, may be configured to display a graphicalrepresentation (e.g., a chart, an object model, or other datarepresentation) that shows relationships (i.e., links 1311) betweenobjects 1326-1338, with each relationship being based on files orcontent located on websites associated with objects 1326-1338. Forexample, files or other content listed in panel 1305 may be selected ordeselected. As files or other content shown in panel 1305 are selectedor deselected, the network of objects 1326-1338 and links 1311 maychange (i.e., appear, disappear, move, and the like). In some examples,files or content may be shown as a network. In other examples, thenetwork representation shown in panel 1307 may be modified by selectingor deselecting files or content listed in panel 1305. In some examples,selection of hosts (i.e., objects 1326-1338) may be varied by selectinga different object in panels 1307 or 1308. Further, relationships (i.e.,links 1311) may also be selected in panel 1307. When an item (e.g.,links 1311, objects 1326-1338) are selected, the display shown in panel1307 may re-center, shift, move, or otherwise be displayed differently.Panel 1305 may be configured to persistently display selected orunselected files by checking or unchecking, respectively, boxes providednext to each file name associated with files or content 1309. When filesor content in panel 1305 are selected, the display in panel 1307 maychange (i.e., objects 1326-1338 may appear or disappear, links 1311 mayvary in configuration between objects 1326-1338, links 1311 may changein numbers, and the like). Alternatively, when the display in panel 1307is modified (e.g., re-centered, shifted, moved, or otherwise modified byselecting, for example, one or more of objects 1326-1338 or links 1311),the list of files or content in panel 1305 may change. Likewise, thelist of files or content shown in panel 1305 may also remain unchanged.Still further, panel 1305 may also be configured to display updates offiles or content based on selections made to items displayed in panel1307 (e.g., objects 1326-1338, links 1311). In other examples, theabove-described panels 1305 and 1307 may be varied and are not limitedto the descriptions provided.

Fingerprints may, in some examples, be displayed in panel 1316, whichmay appear in various forms. Further searches for files, data streams,or content may be performed by entering a fingerprint into panel 1316 oranother panel, which is configured to use a fingerprint as an input forperforming surrogate hashing or surrogate heuristic identification todetermine a value associated with the files or data streams. Any type ofsearch algorithm or technique may be used to compare a data (e.g., astored histogram, fingerprint, combination thereof, or others) providedin panel 1316 to other stored data, for example, in repository 110(FIG. 1) and is not limited to any particular implementation. Panel 1317may be used to upload files for analysis or comparison using surrogatehashing or surrogate heuristic identification. Further, surrogatehashing data or surrogate heuristic identification may be entered (i.e.,input) into any of panels 1304-1317 that may be configured to search forother files, data streams, or content having substantially similaridentifying data or content. For example, a fingerprint may be anidentifier in hexadecimal, binary, ternary, or other types of formatsand is not limited to any type or format. Interface 1300, window 1302,and panels 1304-1317 may be varied in size, shape, function, input,output, and layout, none of which are limited to the examples shown anddescribed above.

Here, address bars 1318-1320 may be configured to allow the input of anaddress, location of a file, or domain. In some examples, address bars1318-1320 may be used to enter a web, Internet address, or otherlocation where a file or data stream is stored. A domain (e.g., a WorldWide Web address, hypertext transfer protocol (HTTP) address, or othertype of address) may also be used as an input to one or more of addressbars 1318-1320. Further address bars 1318-1320 may also be configured toreceive a fingerprint, encoded histogram data, encoded vector data, acombination thereof, or a combination thereof used with other data that,for example, is entered (i.e., using techniques such as “cut-and-paste,”“copy-and-paste,” and others). Once entered, a fingerprint, encodedhistogram data, encoded vector data, a combination thereof, or acombination thereof used with other data may be used to search for othersubstantially similar files, data streams, or other content and, iffound, a file, data stream, or other content may be retrieved andpresented using one or more of panels 1304-1316. For example, if adomain for a website is entered into address bar 1318, the home page ofthe website may be shown in panel 1304. Further, the domain may also beused to show whether a file sought for comparison based on a fingerprintis located at the domain. If found, a copy of a file may also bepresented along with an address, location, or other identifyinginformation that allows a user to determine whether the file, datastream, content, or portion of content is a copy of another file, datastream, content, or portion of content associated with the input data(e.g., fingerprint, encoded histogram data, encoded vector data, acombinations thereof, or a combination thereof used with other data, andothers). As an example, if a rights management agency for an artist issearching for illegally copied versions of a given artist's work, afingerprint associated with the artist's file may be entered into panel1316 and used to search for other files, data streams, or content havingthe same or substantially similar fingerprint. Files, data streams, orcontent with the same or substantially similar fingerprint may be shownin, for example, panel 1308 along with links to other pages within thewebsite or to another website that may have copies of the illegally(i.e., unauthorized) copied file. In other examples, address bars1318-1320 may be configured to input a local address of a file stored,for example, on the same system as, for example, that used to implementapplication 1200.

Interface 1300 and the described techniques may be used to search, find,and otherwise identify files, data streams, or other content. In someexamples, content may be identifiable data found in files or datastreams. In other examples, content may have different file or datatypes, formats, encapsulations, designs, layouts, and the like. Forexample, content may include images, graphics, photos, music, video,audio, and other information. A song, digital recreation of a painting,editorial article, web log (i.e., blog), photograph, and others may becontent. Numerous other types of content may be sought and identifiedusing the described techniques. Further, as described herein, contentmay be information or data that can be discretely sought, found,retrieved, copied, and otherwise identified using, for example,surrogate hashing, surrogate heuristic identification, and othertechniques for identification. Interface 1300 or elements of interface1300 (such as frames 1304-1320) may be used to provide inputs into areporting system (not shown). In some examples, reporting systems ofvarious types, versions, implementations, configurations, or platformsmay be used and are not limited to any specific type of reportingsystem. Interface 1300 may be used as a “front-end” user interface toapplication 1200 and other computer programs, software, or applicationsthat implement the logic, processes, and techniques described herein.Further, interface 1300 and the above-described elements may be variedand are not limited to the examples provided.

FIG. 14 illustrates an exemplary process for using a surrogate hashingor surrogate heuristic identification interface. Here, filters providedusing interface 1300 (FIG. 13) are interpreted (1402). In some examples,filter specifications provide information that determine what criteriaor parameters are used to filter searches for fingerprints performedusing, for example, application 1200 (FIG. 12). Examples of filterspecifications may include geography, domain name, MIME or file type,fingerprints, and others. Once interpreted, a query may be constructed(1404). In other examples, different types of search techniques may beused and modified for, as an example, different data storage facilities,structures, databases, and the like. After constructing a query usinginput received using interface 1300, the query is run (1406). Resultsfrom a query may be loaded, including a fingerprint, a domain, and alocation (1408). Once loaded, a determination is made as to whetherinput has been received using, for example, interface 1300 (1410). Here,a determination is made as to whether rating input has been receivedregarding the search results (i.e., when copies of files or data streamshave been identified, they are loaded into interface 1300 and displayedfor a user to make qualitative or subjective decisions as to whether thefiles or data streams found are copies). In other examples, differenttypes of input may be received apart from ratings (e.g., additionalfiltering criteria, comments or notes, binary actions (e.g., accept,reject), and others). If a rating is received, then the rating is storedin, for example, a repository (1412). If a rating is not received, thenthe fingerprint is skipped (1414). Another determination is made as towhether another fingerprint is available for processing, as describedabove (1416). If another fingerprint is available, then theabove-described process may be repeated. If another fingerprint is notavailable, then the above-described process ends. In other examples, theabove-described process may be varied in function, design, order, numberof steps, or implementation and is not limited to the examples shown.

FIG. 15A illustrates an exemplary fingerprint application. Here,fingerprint application 1500 includes fingerprint module 1504, downloadmodule 1508, and surrogate hash module 1512. Fingerprint application1500 may be implemented on a server, a client, or within anotherapplication (e.g., crawlers installed on servers 104-108 (FIG. 1)). Insome examples, download module 1508 receives or retrieves files or datastream 1514, which is used to generate a fingerprint. When generating afingerprint, surrogate hash module 1512 may be used to hash files ordata streams 1514 using techniques such as those described andreferences above. Further, fingerprint module 1504 generatesfingerprints for content. In some examples, fingerprints are output toclients (e.g., users, user interfaces, and others) and servicerecipients (e.g., application 1200 (FIG. 12)). In other words, clientfiles and service files including the above-described information may begenerated by fingerprint application 1500 and output by fingerprintmodule 1504. Fingerprints may include, for example, a unique referenceidentifier (ID) and a full file name (e.g., an address or other locatinginformation). Fingerprints may also include a reference ID and afingerprint, such as those described above in connection with FIG. 13.In other examples, fingerprints application 1500 and the describedelements may be varied in function, structure, and implementation andare not limited to the examples provided.

FIG. 15B illustrates an alternative exemplary fingerprint application.Here, fingerprint application 1520 may include API 1522, transmissionmodule 1524, fingerprint module 1526, monitoring module 1528, surrogatehash module 1530, and fingerprint data 1532. In other examples, thenumber, type, function, structure, and operation of fingerprintapplication 1520 and elements 1522-1532 may be varied and is not limitedto those shown. Here, data may be input using API 1522 or, for example,transmission module 1524, which may transmit data using various wiredand wireless techniques (e.g., Ethernet, LAN, Gigabit Ethernet,Bluetooth, IEEE 802.11x, and others). Further, monitoring module 1528may monitor or observe files or data streams provided via API 1522 froma crawler, for example. As a monitoring service, fingerprint application1520 may be implemented locally or remotely on a data network to find,catalog (i.e., index), and fingerprint content, files, or data streams.Fingerprints are then stored in fingerprint data 1532. In some examples,fingerprint data may include file names, file paths, machine names, usernames, and other data. In other examples, fingerprint application 1520and the above-described elements may be varied and are not limited tothose shown and described.

FIG. 16 illustrates an exemplary surrogate heuristic identificationsub-process for generating a fingerprint. Here, a file or data stream isreceived (1602). In some examples, a file or data stream may alsoindicate a location, directory, sub-directory, table, or provide otherdata or information that identifies which files or data streams tofingerprint and compare. Once received, files or data streams are thenprocessed as described above, to generate surrogate hashed fingerprintand unique reference IDs for each file or data stream. In some examples,files or data streams may be hashed using surrogate hashing techniquessuch as those described in U.S. patent application Ser. No. 11/408,199,which is herein incorporated by reference for all purposes. Once hashvalues have been generated, then a client file and a service file may beoutput (1606). Client file and service files may include information asdescribed above in connection with FIG. 15A. The above-described processmay be varied in design, function, and implementation and is not limitedto the examples provided above.

FIG. 17 illustrates an exemplary computer system suitable for surrogateheuristic identification. In some examples, computer system 1700 may beused to implement computer programs, applications, methods, processes,or other software to perform the above-described techniques. Computersystem 1700 includes a bus 1702 or other communication mechanism forcommunicating information, which interconnects subsystems and devices,such as processor 1704, system memory 1706 (e.g., RAM), storage device1708 (e.g., ROM), disk drive 1710 (e.g., magnetic or optical),communication interface 1712 (e.g., modem or Ethernet card), display1714 (e.g., CRT or LCD), input device 1716 (e.g., keyboard), and cursorcontrol 1718 (e.g., mouse or trackball).

According to some examples, computer system 1700 performs specificoperations by processor 1704 executing one or more sequences of one ormore instructions stored in system memory 1706. Such instructions may beread into system memory 1706 from another computer readable medium, suchas static storage device 1708 or disk drive 1710. In some examples,hard-wired circuitry may be used in place of or in combination withsoftware instructions for implementation.

The term “computer readable medium” refers to any medium thatparticipates in providing instructions to processor 1704 for execution.Such a medium may take many forms, including but not limited to,non-volatile media, volatile media, and transmission media. Non-volatilemedia includes, for example, optical or magnetic disks, such as diskdrive 1710. Volatile media includes dynamic memory, such as systemmemory 1706. Transmission media includes coaxial cables, copper wire,and fiber optics, including wires that comprise bus 1702. Transmissionmedia can also take the form of acoustic or light waves, such as thosegenerated during radio wave and infrared data communications.

Common forms of computer readable media includes, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, carrier wave, or anyother medium from which a computer can read.

In some examples, execution of the sequences of instructions may beperformed by a single computer system 1700. According to some examples,two or more computer systems 1700 coupled by communication link 1720(e.g., LAN, PSTN, or wireless network) may perform the sequence ofinstructions in coordination with one another. Computer system 1700 maytransmit and receive messages, data, and instructions, including program(i.e., application code) through communication link 1720 andcommunication interface 1712. Received program code may be executed byprocessor 1704 as it is received, and/or stored in disk drive 1710, orother non-volatile storage for later execution.

The foregoing examples have been described in some detail for purposesof clarity of understanding, but are not limited to the detailsprovided. There are many alternative ways and techniques forimplementation. The disclosed examples are illustrative and notrestrictive.

What is claimed:
 1. A system comprising: a memory configured to storedata associated with content, the content comprising multiple,time-ordered portions that each have at least one of audio and videocharacteristics; and a processor configured to analyze the content toidentify the time-ordered portions of the content having at least one ofaudio and video characteristics that meet predetermined criteria not metby other time-ordered portions of the content, to use the at least oneof audio and video characteristics having the predetermined distance togenerate histogram data, the histogram data being used to provideheuristic data, to process the heuristic data to generate a fingerprintfor each portion of the content, and to compare the fingerprint of eachportion of the content against one or more stored fingerprints based onthe predetermined criteria to identify other content that is similar tothe content.
 2. The system of claim 1, wherein the memory is furtherconfigured to store the fingerprint of each portion of the content. 3.The system of claim 1, wherein the content comprises a file.
 4. Thesystem of claim 1, wherein the processor is further configured toprocess the content using a codec.
 5. The system of claim 1, wherein theprocessor is further configured to compare each fingerprint to the oneor more stored fingerprints to identify a location of another contentassociated with the one or more stored fingerprints.
 6. The system ofclaim 1, wherein the portion comprises all of the content.
 7. The systemof claim 1, wherein the processor is further configured to determine avector using the heuristic data.
 8. The system of claim 1, wherein theprocessor is further configured to process the heuristic data and vectordata associated with the heuristic data to determine the fingerprints.9. The system of claim 1, wherein the processor is further configured toprocess the heuristic data using the histogram data and vector dataassociated with the heuristic data without generating a histogram fromthe histogram data.
 10. A computing device comprising: a memoryconfigured to store a file comprising multiple, time-ordered portionsthat each have at least one of audio and video characteristics; and aprocessor configured to: analyze values corresponding to thetime-ordered portions of the file to identify the time-ordered portionshaving at least one of audio and video characteristics that meetpredetermined criteria not met other time-ordered portions of the file;obtain standardized samples from the file and relative to the identifiedportions of the file; generate patterns for the standardized samples;and identify another standardized sample associated with thestandardized samples based at least in part on the patterns.
 11. Thecomputing device of claim 10, further comprising a pattern generatormodule configured to generate the patterns.
 12. The computing device ofclaim 11, wherein the pattern generator module comprises a histogrammodule.
 13. The computing device of claim 10, further comprising asample representation compression module configured to form a compressedrepresentation of the standardized samples.
 14. The computing device ofclaim 13, wherein the sample representation compression module comprisesa hash module.
 15. The computing device of claim 10, wherein theprocessor further comprises a standardized sample selector configured toselect the standardized samples.
 16. The computing device of claim 10,wherein the processor further comprises a sample analyzer configured toanalyze each of the standardized samples to identify a quantityassociated with the at least one of audio and video characteristics,wherein the quantity is used to determine the pattern for acorresponding standardized sample.
 17. A method comprising: storing dataassociated with content, the content comprising multiple, time-orderedportions that each have at least one of audio and video characteristics;analyzing values of at least one of audio and video characteristicswithin the time-ordered portions of the content to identify thetime-ordered portions that meet predetermined criteria not met by othertime-ordered portions of the content; generating histogram data usingthe identified time-ordered portion, the histogram data being used toprovide heuristic data; and processing the heuristic data to generate afingerprint for each portion of the content, wherein each fingerprint ofeach portion of the content is compared to one or more storedfingerprints based on the predetermined criteria to identify othercontent that is substantially similar to the content.
 18. The method ofclaim 17, wherein each fingerprint is compared to one or more storedfingerprints to identify a location associated with other content thatis substantially similar to the content.
 19. The system of claim 1,wherein the processor is further configured to generate a graphical datarepresentation based at least in part on the at least one of audio andvideo characteristic of the identified time-ordered portion, and thegraphical data representation is used to provide the heuristic data. 20.The computing device of claim 10, wherein the standardized samples eachcomprise another portion of the file.
 21. The system of claim 1, whereinthe at least one of audio and video characteristics comprise one or moreof: one or more images, frequency, volume, one or more aural attributes,color, motion, brightness, luminosity, and one or more visualattributes.
 22. The computing device of claim 10, wherein the at leastone of audio and video characteristics comprise one or more of: one ormore images, frequency, volume, one or more aural attributes, color,motion, brightness, luminosity, and one or more visual attributes. 23.The method of claim 17, wherein the at least one of audio and videocharacteristics comprise one or more of: one or more images, frequency,volume, one or more aural attributes, color, motion, brightness,luminosity, and one or more visual attributes.